Location-informed camera

ABSTRACT

A camera ( 90 ) is provided with an arrangement for location-stamping image recordings and with a subsystem for recording location data ( 175 ) independently of making an image recording. The independently-recorded location data is sequentially associated with the image recordings and can be used subsequently to retrieve corresponding photos from a photo archive ( 40 ).

FIELD OF THE INVENTION

[0001] The present invention relates to a camera which can obtainlocation data about its current location and associate it with imagerecordings. As used herein, recording includes image recordings (singlephotographs, sequences of photographs, and video recordings, all whetherchemical or digital), sound recordings, etc.

BACKGROUND OF THE INVENTION

[0002] Classification of photographs, particularly those taken byindividuals and families, has long been a problem due to the effortinvolved in maintaining any organisation of the photos. What is neededis a logical organisation, valid over a lifetime, that requires minimaleffort to implement and is intuitive to use when retrieving photos.

[0003] It has been previously proposed to associate location (andorientation) data with digital photos. The location data can be derivedin any suitable manner such as from a GPS system or by using informationobtained from a cellular radio system. Thus, IBM Technical Disclosure413126 teaches a digital camera provided with a GPS receiver. U.S. Pat.No. 5,712,679 discloses a locatable portable electronic camera which isarranged to send back image and location data when triggered, thelocation data being displayed on a map and the image being shownseparately. Also of interest is U.S. Pat. No. 5,389,934 which describesa portable locating system with a GPS unit that is operative to store atravel history of locations visited.

[0004] Other proposals go further and disclose the management digitalphotographs by using an electronic map to show a thumbnail of eachphotograph at a corresponding position of the electronic map. FIG. 1 ofthe accompanying drawings illustrates the main elements for implementingsuch an system, these element being a digital camera 3 equipped with aGPS receiver for determining camera location using signals fromsatellites 2, a PC 5 for receiving digital photographs 4 downloaded fromthe camera 3 together with GPS-derived location information about whereeach photograph was taken, an album program 6 for managing thedownloaded photographs, a store 7 for storing the digital photographs(plus location information), and a store 8 for storing map data (thestores 7 and 8 will generally be internal to the PC 5 but may byexternal). Such an arrangement is described, for example, in JP10233985A.

[0005] The combination of location-tagged digital photographs withmap-based digital photograph albums should greatly facilitate themanagement of photographs for the ordinary user. However, it isimportant that the photograph management systems provided with themap-based electronic albums are also flexible and easy to use. In thisrespect the above-mentioned JP 10233985A describes the possibility ofthe user classifying each photograph whilst JP 8335034A discloses theuse of an icon to represent groups of photographs on a map display.

[0006] It is an object of one aspect of the present invention to furtherfacilitate the making of collections of photographs (and other imagerecordings) by augmenting sets of taken image recordings.

[0007] Certain arrangements described hereinafter utilize data bearerservices and location services of cellular radio networks. Such networksare widespread though the afore-mentioned services are only now beingtaken up significantly. To facilitate an understanding of the describedarrangements, a brief review of cellular network technology and how theforegoing services can be implemented is given in the Annex appearing atthe end of this description, reference being had to FIG. 2 of theaccompanying drawing that depicts a mobile location-aware service beingprovided to a mobile entity 20 via a Public Land Mobile Network (PLMN)10 .

SUMMARY OF THE INVENTION

[0008] According to the present invention, there is provided a digitalcamera comprising:

[0009] an image recording subsystem for making image recordings;

[0010] a location-providing subsystem for providing location data itemsrepresentative of the current location of the camera,

[0011] a location-stamping subsystem for associating with imagerecordings taken using the image recording subsystem, respectivelocation data items, provided by said location-providing subsystem, thateach represent the location of the camera at the time the correspondingimage recording was taken, and

[0012] a location recording subsystem for effecting user-triggeredrecording, independently of the recording of an image recording, oflocation data items provided by the location providing subsystem, theselocation data items and those associated with the location-stamped imagerecordings being recorded in a sequential association with each other.

BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Embodiments of the present invention will now be described, byway of non-limiting example, with reference to the accompanyingdiagrammatic drawings, in which:

[0014]FIG. 1 is a diagram of a known map-based photographic albumsystem;

[0015]FIG. 2 is a diagram of a known communications infrastructureusable for transferring voice and data to/from a mobile entity;

[0016]FIG. 3 is a diagram of an electronic photographic album systemshowing the five operating modes of an album program;

[0017]FIG. 4 shows the fields of a photo record and group record of theFIG. 3 system;

[0018]FIG. 5 shows state data items maintained by the album program ofFIG. 3;

[0019]FIG. 6 shows a typical display output during a “Catalogue”operating mode of the FIG. 3 system;

[0020]FIG. 7 shows a typical display output during a “Map View”operating mode of the FIG. 3 system;

[0021]FIG. 8 shows a typical display output during a “Photo Show”operating mode of the FIG. 3 system;

[0022]FIG. 9 is a diagram illustrating the transfer of location datafrom a cell phone to a digital camera;

[0023]FIG. 10 is a state diagram for the location-data transfer processof FIG. 6;

[0024]FIG. 11 is a diagram illustrating the transfer of camera imagedata via a cell phone and a PLMN to a PC running the FIG. 3 albumprogram, the image data being location stamped with the cell phonelocation during transfer;

[0025]FIG. 12 is a diagram showing the independent transfer of imagedata and location data to a PC running the FIG. 3 album program;

[0026]FIG. 13 is a state diagram of a location log function of a mobileentity equipped with location discovery means;

[0027]FIG. 14 is a diagram illustrating the main steps of the Load andCatalogue operating modes of the FIG. 3 album program in the case ofimage data and location data being separately provided;

[0028]FIG. 15 is a diagram illustrating the matching of location data toimages by matching patterns of timestamps;

[0029] FIGS. 16A-D show a user-effected correction of mismatchedsequences of images and location data; and

[0030]FIG. 17 is a diagram showing the recording of the location ofdesired but not taken photos and the subsequent retrieval of matchingimages.

BEST MODE OF CARRYING OUT THE INVENTION

[0031]FIG. 3 depicts a photo system in which a digital camera 3 providedwith location determining means (such as a GPS receiver) is used togenerate digital photos 4, each photo (also referred to as ‘image data’)4 being stamped with location data indicating where the photo was taken.Other data may also be associated with each photo, such as a timestamp,a camera ID, and a user ID; such associated data (including the locationdata) is herein referred to as photo meta data. The photos and theirmeta data are downloaded by any suitable means (USB connection,removable storage device, etc) into a PC 5 where an album program 50serves to store the photos in a photo store 7 and the photo meta data ina meta-data database 9 (each photo and its meta data being linked by asuitable key associated with both). The album program also has access toa map store 8. The stores 7 and 8 and the meta-data database can be onthe PC or external.

[0032] The album program enables users to catalogue, manage and viewtheir photos through a map-based interface, the photos being representedon a displayed map by a marker indicating the location they were taken.

[0033] More particularly, the album program comprises five mainoperating modes 51 to 55 and a user can move between these modes or quitthe program by making an appropriate choice (for example, by soft keysdisplayed on the PC display). FIG. 3 indicates for each mode the mainchoices available to the user (for example, the label “View” in Start UpMode block 51 indicates that the user can choose to change to the ViewMap Mode 54. The role of each operating mode is as follows:

[0034] Start Up Mode 51—This is the initial mode upon start up of thealbum program and it permits a user to select either the Load Mode orthe Map View Mode, or to quit the program.

[0035] Load Mode 52—In this mode, the user can download data from camera3; when the user has finished, he/she indicates this (see the “Done”label) and the mode changes either to the Catalogue Mode 53 if anyphotos have been loaded, or back to the Start Up Mode 51 if no photoswere loaded.

[0036] Catalogue Mode 53—In this mode, the user can manage newly loadedphotos using a map based display, this management including assigningthem to one or more groups (sets of related photos). From this mode, theuser can move back to the Load mode to load more photos or to the ViewMap mode for browsing the photo album; the user may also choose to quitthe program.

[0037] Map View Mode—The Map View Mode 54 is the mode in which a usercan browse the album and select photos for viewing. Browsing is on thebasis of displaying maps with the location of photos indicated. From theMap View 54, a user can move to the Load, Photo Show, or Catalogue Modesor quit the program.

[0038] Photo Show Mode—In this mode, the user can view a photo selectedin the Map View Mode; the user can also step through a series of relatedphotos. From the Photo Show Mode, the user returns to the Map View Mode.

[0039]FIG. 4 shows the meta data record 56 held in database 9 for eachphoto, it being appreciated that some of the fields may be empty for anyparticular photo. The fields comprise:

[0040] Album ID—This is a unique identifier for the album.

[0041] Camera ID—This is a camera identifier that may be either suppliedautomatically by the camera in the photo meta data or added by the userwhen downloading photos.

[0042] User ID—This is a user ID which again may be either suppliedautomatically by the camera in the photo meta data or added by the userwhen downloading photos

[0043] Photo ID—This is a unique photo ID provided by the album programand can conveniently be made up of a load batch number (a new batchnumber being assigned for each session of downloading data from acamera) and a number-in-batch identifying the photo from others in thesame batch.

[0044] Accession Date—This is the data of loading of the photo by thealbum program (photos in the same batch will have the same accessiondate).

[0045] Location Data—The location data provided with the photo by camera3.

[0046] Date/Time Taken—The timestamp data provided with the photo bycamera 3.

[0047] Short Title—A short descriptor of the photo provided by the user.

[0048] Description—A fuller user-provided description of the photo.

[0049] Semantic Loc.—A user-meaningful location description (e.g. EiffelTower) as opposed to the coordinates provided by the location data. Thisfield overlaps in intent with the two preceding fields and is optional.

[0050] With respect to user ID, where this is supplied automatically bythe camera, the user ID will have been set into the camera at some stageby the user. The camera can be provided with suitable means for enablingeach of several users to set in their ID at every usage and/or the meansfor enabling several different users to set in and store their IDs witheach such user selecting their ID from the stored IDs each time thatuser starts to use the camera; input of ID data can conveniently be doneby transfer from a computer thereby avoiding the need for an inputkeypad associated with the camera. Alternatively, the camera can bepre-programmed with a set list of identifiers (numbers, icons, colours,animal types, etc) and users choose which identifiers to employ todistinguish amongst them; in this case, the camera simply needs to beprovided with input means for enabling a user to select their identifierfrom the programmed list of identifiers. Thus, a camera intended forfamily use may have pre-programmed animal icons as identifiers with themother and father choosing, for example, icons of a dog and cat andtheir three children choosing lion, tiger and rabbit icons respectively.Of course, to handle cases where icon identifiers are used, the albumprogram is preferably adapted to store and manage user IDs of this form.

[0051] The database 9 also holds group records 57 each relating to agroup of user associated photographs. Each group record comprises agroup name, an indicator of a group icon to be used to represent thegroup, a brief description of the group, and a linked list of the IDs ofphotos making up the group. A photo can reside in non, one or moregroup.

[0052]FIG. 5 shows the main state data items 59 maintained by the albumprogram so that it knows what operational point it has reached and howto restore itself to certain prior conditions as necessary. These dataitems comprise:

[0053] Current Mode—The current operating mode.

[0054] Previous Mode—The previous operating mode, if any. This isrequired when about to enter the Map View mode from the CatalogueMode—if the user is merely returning to the Map View Mode after havinggone into the Catalogue Mode (for example, to change meta data about aphoto), then the user will expect to reurn to the same map and displayas before whereas if the user is entering the Map View Mode from theCatalogue Mode after having catalogued a newly loaded set of photos, theuser will have different expectations.

[0055] Current Photo—This is the currently selected photo, if any. Theselected photo may be one currently be displayed in full or merelyrepresented by an icon or thumbnail.

[0056] Current Batch—The batch number of the current batch of photos or,where photos from more than one batch are being examined, then the batchnumber of any currently selected photo.

[0057] Current Group—The currently selected group, if any.

[0058] Filter Details—Details of any filter being applied to theselection of photos in the Map View and Photo Show Modes.

[0059] Most recent:

[0060] Map View map—The most recently viewed Map View Mode map(including area displayed)

[0061] Catalogue map—The most recently viewed Catalogue Mode map(including area displayed)

[0062] Other features of the album program will be described below aspart of the description of a typical sequence of operation.

[0063] A user with photos to download to the album program starts theprogram and chooses the “Load” option presented by the Start Up Modeprogram code. The Load Mode interacts with the user to enable thedownloading of photos and meta data from the camera 3, photo ID data(Batch and number-in-batch) and accession date being automatically addedby the program to the meta data of each photo. The user may also beenabled to add in user ID data for the whole batch, overriding any suchdata coming from the camera. Upon termination of loading, the userselects “Done” and the album program automatically progresses to theCatalogue Mode to enable the user to carry out cataloguing functions inrelation to the newly-loaded batch of photos.

[0064] In the Catalogue Mode, the album program generates a display ofthe form shown in FIG. 6 comprising a central map area 61, left andright margin areas 62A, 62B and upper and lower control bars. The mapdisplayed in map area 61 is sufficient to encompass the locationsregistered for the newly loaded batch of photos. A thumbnail 63 of eachnew photo is shown in one or other of the margin areas 62A, B and a leadline 65 connects each thumbnail 63 to a corresponding marker 64 showingon the map the location where the photo was taken. The use of margins toshow the thumbnails and lead lines to indicate the corresponding maplocations is preferred as being less cluttered than trying to place thethumbnails directly in the correct locations on the map.

[0065] The upper control bar comprises three controls 66, 67, 68 thatprovide access to the following functionality, generally in respect of apre-selected photo (this pre-selection being effected by clicking on thecorresponding thumbnail):

[0066] Show Photo Control 66—this displays the photo corresponding to aselected thumbnail 63, with return to the Catalogue map being effectedby a mouse click;

[0067] Edit Photo Details Control 67—this displays the record details 56of a selected thumbnail and enables editing of these details;

[0068] Group Control 68—this control permits groups to be created, and aphoto to be assigned to one or more groups (the photo ID being addedinto the group record 57).

[0069] The group control comprises a drop-down box 68A operated bycontrol element 68B, the box normally displaying a currently selectedgroup, if any. The user may cause the box to drop down (by clicking oncontrol element 68B) to show a list of available groups from which theuser can select one, this list also including an option to create a newgroup. Selecting this latter option takes the user to a creation screenwhere the user enters details of the new group. The details of acurrently selected group can also be brought up for editing by operating(clicking on) control element 68C. To assign a photo to the currentgroup, the corresponding thumbnail is selected and then the “Apply”control element 68D is operated. Preferably, double clicking the “Apply”control gives the mouse cursor the “Apply” power so that any thumbnailselected using the cursor is assigned to the current group (clicking onthe Apply element again removes this power).

[0070] By way of example, a user may decide to create a group for photostaken on a particular holiday or a group for all photos related to acurrent or past abode. The group icon can be selected from a set ofavailable icons or created by the user. Thus a current-abode group mayhave a house icon as its group icon whilst a holiday in London may berepresented by a Tower Bridge icon.

[0071] The lower control bar includes a zoom control 69 that enables auser to zoom in or out around a particular point on the displayed map.More particularly, to zoom in on a target point, the “+” element ofcontrol 69 is selected, the display cursor placed on the target pointand clicked. The “+” and “−” elements effect stepped, progressive,zooming; in contrast, the “Full In” element goes straight to closestzoom on a target point whilst “Full Out” returns to the original mapdisplay encompassing all of the newly loaded photos.

[0072] Controls 70, 71 and 72 respectively enable a user to change tothe Load Mode, change to the Map View Mode, and Quit the program.

[0073] The album program is preferably operative to accept photos forwhich there is no meta data, including no location data. In this case,the corresponding meta data record initially only contains thealbum-generated data (Photo ID, accession data), and the Catalogue Modeis arranged to represent these photos but without lead line or locationmarker until such a time as the user enters location data into thelocation field of the photo record 56 either directly or by a facilityfor adding this data by pointing to a location on the map display.

[0074] Once a user has finished editing the photo meta data andassigning the photos to groups, the user may decide to browse the albumand accordingly operates the “View” control 71. FIG. 7 depicts a typicalMap View Mode display; for ease of understanding, the same referenceshave been used on corresponding elements appearing in the Catalogue Modeand Map View Mode.

[0075] The starting map displayed in the Map View Mode is, for example,a world map or a map encompassing the locations of all the photosrecorded in the album; alternatively, the map displayed could be thesame map as being used in the Catalogue Mode before transiting to theMap View Mode. Initially, all photos relevant to the starting map willbe represented either as thumbnails 81, individual photo icons 80, groupicons (see current-abode group icon 85), or concentration icons (seeicon 82). A concentration icon represents a number of photos that arenot in a common group but were taken in the same general area and cannotbe individually represented at the current map display resolution; thearea concerned can be represented by a bounding circle 83. Where aconcentration icon only encompasses photos that belong to a commongroup, the concentration icon is replaced by the group icon. Similarly,where a concentration icon encompasses at least a threshold number (e.g.5) of photos that belong to a common group but other photos as well,then the group icon is shown alongside the concentration icon. If thethreshold is crossed for several groups then each group icon will beshown (in determining whether the threshold is crossed, if a photobelongs to more than one group, it is counted towards the threshold foreach group).

[0076] Zooming in and out is controlled by the same control 69 asalready described for the catalogue Mode. If zooming in permits thephotos of a concentration to be represented individually then this isdone, the photos being collapsed back to a concentration icon and/orgroup icon on zooming back out. Scrolling the map display left, right,up or down is effected by scroll control 75 (or else scroll bars can beused).

[0077] To minimise clutter, a user may set an option (through a“customise menu”, not shown) by which all photos of a group areinitially represented by the corresponding group icon even if there isroom to display the thumbnails of the group photos encompassed by thecurrently displayed map. In this case, the group icon is displayed withits location marker at the location of a user-specified one of thephotos of the group (to implement this, a further group-control element“set leader” can be provided which, when activated, would store thephoto ID of a currently-selected photo into an extra field of the grouprecord of the current group, the location of this photo being the“location” of the group). To give access to the individual photos of agroup, one or both of the following mechanisms can be used:

[0078] Single clicking on a group icon brings up a scrollable list ofthe photos in the group (preferably with date taken information andshort title). Mouse rollover of a photo entry in the list causes thelocation marker of that photo to be displayed on the Map View (thephoto's thumbnail can either be shown in the scrollable list all thetime the list is displayed, or adjacent its location marker but onlywhilst the latter is displayed); moving the mouse cursor off the entrycauses the location marker to disappear. Clicking the list entry causesthe thumbnail to be inserted into a margin area with a lead line to acorresponding location marker on the map, this thumbnail being retainedfollowing closure of the group photo list. A variant of the above is tohave display of the list accompanied by display of location markers forall the group photos encompassed by the current map—rollover of a listentry would then highlight the relevant location marker.

[0079] double clicking on a group icon unpacks the group and causes itsphotos to be represented as individual photos (or in concentrations).

[0080] Generally, whenever a particular photo is selected (for example,by clicking on it) the name of the related group (if any) is displayedin the group control element 68A—if a photo belongs to more than onegroup, these groups are viewable by opening up the dropdown box usingcontrol 68B. The details of the selected photo can then be viewed (butnot edited) by operating control 78 or the photo viewed by operatingShow Photo control 74 which causes the mode to switch to the Photo ShowMode. The details of the current group, if any, can be viewed byoperating the view group control 68E.

[0081] Filter control 76 enables a user to select which photos are to berepresented on the map display. Selection criteria can, for example,include one or more of the following:

[0082] Date range of when the photo was taken;

[0083] User ID

[0084] Camera ID

[0085] Group name

[0086] Batch ID

[0087] Accession date

[0088] Key word in short title/description/semantic location.

[0089] Access to a particular's user's photos can be password protected

[0090] If the user wishes to edit the details of a photo or group, theuser must select the relevant photo and return to the Catalogue Mode;the map and photos displayed in the Catalogue will be those of the samebatch as the selected photo.

[0091]FIG. 8 shows the Photo View Mode display brought up by clickingthe Show Photo control 74 in the Map View Mode when a photo is selected.A full size image 79 of the photo is displayed and the user can view thephoto and group details using the controls 67 and 68 respectively.Furthermore, a control 77 permits the user to view related photos in thesame group (if photo is in more than one group, this will be the groupappearing at the top of the dropdown box 68A, a different group beingselectable by dropping down the group list); the group photos areaccessed, for example, in date/time of taking order. If a photo is notassociated with a group, then the album program permits photos of thesame batch to be viewed, ordered by number.

[0092] The provision of suitable program code for implementing theabove-described event-driven functionality is within the competence ofpersons skilled in the art.

[0093] A similar map-based album to that described above can also beused to classify and access other types of recording such as soundrecordings, video recordings etc. Where the data is non-visual, thethumbnails and full-sized photo image representations of theabove-described electronic photo album will be replaced by thecorresponding representations for the recording concerned.

[0094] Uniting Location and Recording Data—at the time of generation

[0095] Of course, the vast majority of current cameras are not providedwith location determining means. Nevertheless the foregoing map-basedalbum can still be built up provided the user can activate a locationdetermining device whilst located at the place a recording is being/hasbeen/is about to be, made. In the near future, many location-determiningdevices (such as GPS devices) will be widely deployed; potentially moresignificantly, location services will become widely available to usersof mobile phones (see the Annex to this specification which describesthe mobile radio infrastructure and the provision of location-basedservices using such an infrastructure).

[0096] Thus it will become relatively easy for someone taking a photo tofind out their location using their mobile phone. However what isadditionally needed is some way of uniting this location informationwith the photographs.

[0097] One way of doing this is illustrated in FIG. 9 where a digitalcamera 90 is provided with a communications link to receive locationdata from a mobile entity 20 (here shown as a mobile phone, by way ofexample). More particularly, camera 90 comprises optics 91, sensor array92, image processing block 99, control block 93, memory 94 for storingphoto image data 95, and a communications interface 96. Cell phone 20comprises, as well as its radio subsystem 22, a data handling subsystem23, and communications interface 97. Interfaces 96 and 97 arecompatible, enabling the camera 90 and cell phone 20 tointercommunicate; interfaces 96 and 97 are, for example, suitable forestablishing an infrared or short-range radio link between the cameraand cell phone.

[0098] Cell phone 20 also includes location-discovery means 29 by whichthe cell phone can ascertain its location, this location discovery beingeffected when control 28 (hard or soft button) is operated by the user.The location discovery means is, for example, a program run by the datahandling subsystem for requesting location information from a locationserver of the mobile radio infrastructure; however, the locationdiscovery means could alternatively be a GPS system built into the cellphone. Whatever form the location discovery means takes, when button 28is operated, location data 98 is generated and is available in the phonefor transfer to the camera 3.

[0099] The data handling subsystem runs a transfer program fortransferring the location data over a link establish between theinterfaces 96, 97. The control block 93 of the camera is operative toreceive this location data and associate it with the last-taken photo.FIG. 10 shows a top-level state diagram of how this process is managedby association functionality of control block 93. Normally theassociation functionality resides in a state 100 in which it is ready toreceive location data through interface 96; whilst in this state, thecamera can be used to take photographs and the association functionalityremains in state 101. However, upon location data being passed from cellphone, the association functionality transits to state 101 in which thecamera is blocked from taking a photograph. In state 101, theassociation functionality of control block 93 receives the location dataand associates it with the last taken photo. Once this is done (and itgenerally will happen very rapidly) the association functionalityreturns to state 100.

[0100] Of course, it would also be possible to have the taking of thephoto by camera 90 trigger the location discovery by the cell phonefollowed by transfer to the camera.

[0101] Another way of uniting a digital photo and location data isillustrated in FIG. 11 and involves uploading the photo image data 95through the cell phone (via a link established between camera 90 andcell phone 20 through interfaces 96 and 97), to a network store 43 of aservice system 40 (arrow 105 represents this transfer). The servicesystem 40 resides either in the mobile infrastructure or is accessiblevia the latter over a data-capable bearer service. En route to thestore, or upon loading into the store, location information 98 on themobile phone is requested and associated with the photo image data 95;in the first case, the location data is obtained by the cell phone andassociated with the image data as the image data is being transferred tothe store 43 whilst in the second case, a control function 42 of thestore is operative to request the location data 98 from location 41immediately upon the image data being received by service system 40. Ofcourse, this method will generally need to be effected for each photoimmediately it is taken since otherwise the location of the cell phonemay not correspond to the location where the photo was taken.

[0102] The foregoing methods of associating separately generated imageand location data at around the time of generation can equally beapplied to other types of recording.

[0103] Uniting Location and Recording Data—subsequent to when generated

[0104] In many cases, it will not be possible, for whatever reason, tolink the camera 90 with a cell phone or other available locationdiscovery means (such as a stand-alone GPS device). For these cases, alocation log can be created for subsequent correlation with the photosbeing taken. More particularly,

[0105] as the camera is used to take a number of photographs, thephotographs as items are distinguished from each other by an implicit(e.g. sequence position) or explicit location-independent referenceassociated with each;

[0106] in association with taking each of at least some of saidphotographs, a mobile device that is separate from the camera and iscapable of effecting or triggering location discovery of its position,is used to generate location data indicative of the location at whichthe photograph was taken, this location data being stored together withan index matching the reference associated with the correspondingphotograph;

[0107] subsequently, the location data is united with the correspondingphotographs by a correlation process using said references and indexes.

[0108] The mobile device is, for example, a cellular-radio-based mobiledevice (phone or e.g. a PDA with mobile radio capability) capable ofeffecting location discovery such as by requesting location data from alocation server; the mobile device may take other forms such as astandalone GPS device.

[0109] References can simply be position-in-sequence of photographs (inwhich case the corresponding indexes are similar ordering data).Alternatively, the references can be timestamps—in this case, theindexes could be timestamps also (or, again, ordering data sincetimestamps are also this).

[0110] The photos can be traditional (chemical) snaps and the uniting isdone by printing labels with the location data, these labels then beingstuck on the back of the snaps (preferably this location data takes theform of a map showing the location where the photo was taken)—in thiscase, the labels are numbered to correspond to photo numbers.

[0111] Preferably, however, the photos are digital (or digitised) andthe uniting of the photos with the location information is done in a PCor other computing device as part of the album program. Processes foreffecting this uniting will be described hereinafter.

[0112] With regard to how the location data is transferred to the samecomputing device as the photo image data, a number of possibilitiesexist and FIG. 12 illustrates three such possibilities in the case wherethe mobile device is a cell phone 20. More particularly, FIG. 12 shows acamera 3 and cell phone device 20 both possessed by the same user. Thecell phone 20 communicates with mobile radio infrastructure 10. Whenevera user takes a photo he/she operates a button 28 of the cell phone tocause the cell phone to trigger a determination of its location eitherby itself or through location server 41 of the PLMN 10. A log oflocation data on each photo taken is built up. In due course the usertransfers the image data 95 from the camera 3 to computer 5 running thealbum program 50. As regards transfer of the location log to thecomputer, the following three possibilities are illustrated:

[0113] a)—location data for each photo is accumulated in a location log100 stored in the cell phone and subsequently transferred (see arrow111) directly to the computer 5 over a wire link, infrared link orshort-range radio link.

[0114] b)—location data for each photo is accumulated in a location log100 stored in the cell phone and this log is subsequently transferred(arrow 109) via a data-capable bearer service of the PLMN 10 to a store47 (in the PLMN or a connected network, such as the Internet). Thelocation log is later retrieved by computer 5 from store 47 (see arrow110).

[0115] c)—Operation of cell-phone button 28 sends a request (arrow 107)to a log-service controller 44 of a log server system 40 to obtain thelocation of cell phone from location server 41 and store it in a log 100held in store 45 of the service system, the identity of the log to beused being included in the request. The log 100 is subsequentlyretrieved by computer 5 from store 45 (see arrow 108).

[0116] The same processes as described above can being effected forother types of recordings, the location data being separately determinedand subsequently re-united with the recording concerned. In the case ofa sound recording done, for example, on a tape recorder, the locationdata could even be provided by a digital camera equipped with GPS.

[0117] It may be noted that giving a mobile phone the ability to store alocation log (either in the phone itself or in the mobile infrastructureor in a connected network) is itself a useful feature. Thus whilst manylocation-based services simply require a one-off provision of locationdata or continually monitor location, the ability for a user toselectively trigger location determination for storing the resultantdata to a log has value in its own right—for example, a user may wish tostore the location of places visited whilst out walking or, as describedabove may want to log the locations of photos taken. Since the user mayalso want to use other location-based services at the same time, theuser must be able to select when location information is to be logged.Further, since the user may want to log location information aboutdifferent topics, the mobile phone (or otherlocation-determination-triggering device) preferably permits a user toset up more than one log at a time and to select to which log aparticular item of location data is to be stored.

[0118]FIG. 13 shows a controlling state machine for a location-logapplication capable of managing multiple location logs, the applicationrunning, for example, on a data handling subsystem of a mobile entity(such as a cell phone) that has means for discovering its locationeither directly or from a location server. Selection of the applicationsets the application into a menu state 120 that presents the user withthe choices of creating a new log, using an existing log ,or uploadingan existing log (for example, to network store 47 or computer 5 in theFIG. 12 arrangement). If the user chooses to create a new log, state 121is entered in which the user is asked to specify certain details aboutthe log (in particular, its name); in due course new log 100 is createdand the log application automatically transits to state 124 in whichlocation can be added to the log. This same state 124 is also reachedwhen the user chooses the ‘use existing log’ option from the openingmenu, the log application first entering state 122 in which the userselects from a list of existing logs, the log to be used; selection ofthe log to be used automatically moves the log application to state 124.

[0119] When in state 124, the log application responds to an externaltrigger to add a location to the currently-selected log, by obtainingthe current location of the mobile entity and logging it to thecurrently selected log together with a timestamp. The log applicationcontinues in state 124 with the same log selected until the user eitherquits the application or chooses to return to the menu state 120. Theexternal trigger for adding a location can either be user input (e.g. byoperating a hard or soft button) or a command received from anotherdevice. Because the log application initiates location-data requests tothe location providing means of the mobile entity, it is straightforwardto arrange that the log application is only passed relevant locationdata (that is, location data it has requested) and therefore it will noterroneously log location data provided for other applications.

[0120] If the user chooses the upload option from the menu state, thelog application transits first to a selection state 123 in which theuser selects the log to be uploaded and then to an upload state 125. Inthe upload state the log application oversees the transfer of theselected location log. Upon completion of transfer, the log applicationreturns to the menu state 120.

[0121] Recording a location independently of taking a photo still hasrelevance to photo creation. For example, the situation may arise that auser would like to take a photograph of a place or item but has run outof film/on-camera storage, or is present at a time when it is notpossible to take a photograph (at night, in heavy rain or mist, etc). Insuch cases, the user can record their location in their photo locationlog and subsequently retrieve from the Web (or other photo archive) aphotograph similar to that the user wanted to take.

[0122] Where a camera is provided with location discovery means 29 forlocation stamping photos (see camera 90 in FIG. 17), the control means93 of the camera, when activated by user operation of input control 98,can be arranged to enable additional location information 98 to bestored in memory 94 without the need to actually record image data 95;this permits the camera to log the location of desired but untakenphotos. The location data that is recorded independently of taking aphoto (‘independent location data’), is preferably stored in sequencewith location data associated with photos actually taken(‘recorded-photo location data’). Thus, for example, the independentlocation data can be treated as a normal ‘image+location’ data item withzero image data (see item 175)—in other words, both the location-stampedimage data and the independent location data items, are recorded in datastructures of the same form, the data structures being arranged inmemory or explicitly linked into a sequence indicative of the order ofrecording. Alternatively, the independent location data can be stored inits own log, separate from the recorded-photo location data, along withsequence data indicating its sequence position relative to the sequenceof recorded photo data.

[0123] Whilst separate user controls can be provided for triggeringtaking of a photo and for triggering the recording of independentlocation data, in a preferred implementation image recording andindependent location recording are both triggered using the same commonuser-operable control; in this case, a sensor is provided fordetermining whether there is an image to be recorded (if there is, thena location-stamped image recording is made whereas if no image ispresent, an independent location data recording is made). This sensor isfor example, a sensor for detecting whether a lens shutter of the camerais open or closed; alternatively, the sensor takes the form of anoptical sensor located in an image plane of the camera (where the cameraincludes an electronic image capture device such as element 92 in FIG.17, then this device can be used as the image-presence sensor).

[0124] Matching Separately-Generated Image and Location Data

[0125] In order to accommodate the separate provision of image data andlocation data, the album program 50 described above with reference toFIGS. 3-8 is adapted as depicted in FIG. 14. More particularly, the LoadMode is adapted to independently load the image data and the locationdata (block 141), the data loaded from the camera being handled asbefore but without the location data field being filled in on each photometa-data record 56 whilst the location data is temporarily stored in alog identified as related to the batch of photos concerned.

[0126] The Catalogue Mode is now split into two operating phases in thefirst of which the image data and location data are correlated (blocks142 to 144), the second phase being the grouping and details-editingstage that formed the original Catalogue Mode. With regard to the firstphase, this involves an automatic correlation process (block 142),followed by a user-directed correlation adjustment process (block 143);the resultant correlation of image and location data is then committedfor storage by the user (block 144) at which time the location datafield of each photo meta-data record is updated and the separatelocation log deleted.

[0127] In the event that the location log includes desired-but-not-takenphoto location data, there is an additional process (see dotted block146) between blocks 143 and 144 in which the user is given the option offetching (or initiating an automatic fetch of) photo image data from theInternet to match the location concerned. This process is depicted inFIG. 17 where desired image data is supplied (arrow 172) by aspecialised service 174 set up to provide such image data in response torequests (arrow 171). Preferably, where automatic fetching isimplemented, more than one photograph will be retrieved on the basis oflocation, the user then being presented with a choice of third-partyphotos to add to the user's own photo album. As a preliminary step tofetching one or more photographs, the user can be presented with adetailed map 147 of the area around the desired-but-not-taken photolocation 148—the user can then specify approximately what subject/view149 they are interested in (the location data by itself not indicating,for example, the direction in which the user was looking when thelocation was logged or whether the user was interested in a near fieldobject or a far view). The user can specify the view of interest by, forexample, clicking a target point or defining a target area on the mapdisplay. The information derived from the user is passed with therequest for retrieving relevant photos.

[0128] The user may, in fact, decide to defer fetching image data untillater in which case the act of committing the existing correlation inblock 144 also causes the creation of a photo meta-data record for thedesired-but-not-taken photo and such ghost photos will be subsequentlyrepresented in the displays of FIGS. 6 and 7 by “?” icons; clicking onsuch an icon can be arranged to initiate, at least in the CatalogueMode, the process for fetching an appropriate image.

[0129] Considering now the automatic matching process of block 142, oneefficient way of doing this is by time-stamping digital photos in thecamera and time-stamping the location data that is separately created atthe same time (approximately) in a different device. Because differentclocks are used for the two time stamps, absolute time is not reliablefor matching the location data with the photo image data. However, thepattern of timestamps (i.e. time-interval related data) can be used toperform a match. This remains true even when there are additionalentries in either the batch of photos or the location log that have nocounterpart in the other collection. FIG. 15 shows an example in which atimestamp sequence 150 of a batch of eight photos is to be matchedagainst a timestamp sequence 151 of a location log with seven entries.The individual photo timestamps are represented by marks 152 whilst theindividual location timestamps are represented by marks 151. As can beseen, it is a relatively easy matter to match up the two patterns oftimestamps notwithstanding that there are two time-stamped photos 154for which there are no corresponding location entries and onetime-stamped location 155 for which there is no corresponding photo(this may be because the location corresponds to a desired-but-not-takenphoto location). Appropriate pattern matching techniques for effectingthe automatic matching of the timestamp sequences 150, 151 are wellknown to persons skilled in the art.

[0130] The same approach could be used for matching other types ofauxiliary data (and not just location data—for example sound clip data)with photos; again, the matching process can be used with any type ofrecording, not just photos.

[0131] As already noted, matching can also be done on the basis ofsequence number and this can be done even where the photos are onlyphysical items—in this case, the location data is printed out onnumbered self-adhesive labels than can be stuck to the back of thecorresponding photos.

[0132] Returning to matching location data and photos in the albumprogram 150, whilst using sequence numbers, for example, seems an easyway to match up a set of photos with a corresponding set oflocation-data items, it is quite likely that there will beadditions/omissions in one set as compared to the other. As a result thematch between the sets will be imperfect. Mismatching may also arisewhere other correlation keys (that is, not sequence position) are used.However, it may generally be assumed that the ordering of entries is thesame for both sets.

[0133] To correct the match up, a user must intervene and manuallycorrect erroneous associations between entries in the two sets—thisbeing the purpose of the process represented by block 143 in FIG. 14.This adjustment process can conveniently be done by generating aCatalogue Mode display such as shown in FIG. 6 on the basis of thematching achieved after running the automatic match process of block 142(where implemented), or else simply by pairing off photos withlocation-data items in sequence order until one of the sets (photos;location-data items) runs out. In the resultant display, lead lines 65connect photo thumbnails 63 with location markers 64 on the map. Tocorrect an erroneous association, a user drags the map end of therelevant lead line 65 to the correct location marker 64 on the map—ordrags the photo end of the lead line to the correct photo (or simplyclicks on the matching entries in turn).

[0134] To minimize the number of times this needs to be done, use ismade of the consistency of the ordering of both sets—in particular, theassociations of photos and location data for entries later in theorderings than a just-corrected association, are re-matched taking intoaccount the corrected association. If these entries include an alreadycorrected association, this latter is not disturbed. This feature isillustrated in FIGS. 16A-D where:

[0135]FIG. 16A—shows an initial matching of a set of photos 160 with aset of location-data items 161, the photos and location-data-items beingpaired off until the location-data item set is exhausted.

[0136]FIG. 16B—user determines that the third location-data item 165 isactually associated with the fifth photo 166 and corrects theassociation accordingly; this results in a re-pairing of alllocation-data items subsequent to the item 165 with photos subsequent tophoto 166 as illustrated.

[0137]FIG. 16C—similarly, user determines that the seventh location-dataitem 167 is actually associated with the tenth photo 166 and correctsthe association accordingly; this results in a re-pairing of alllocation-data items subsequent to the item 167 with photos subsequent tophoto 168 as illustrated.

[0138]FIG. 16D—user now decides that the second location-data item 169should be associated with the third photo 170 and corrects theassociation accordingly; no consequential downstream adjustments aremade since the next association is one previously established by theuser (between location data item 165 and photo 166).

[0139] It will be appreciated that the match-adjustment processdescribed above with reference to FIG. 16 can be used to associatelocation data with other types of recordings.

[0140] ANNEX A—Mobile radio infrastructure; Location Determination

[0141] This Annex forms an integral part of the specification.

[0142] Communication infrastructures suitable for mobile users (inparticular, though not exclusively, cellular radio infrastructures) havenow become widely adopted. Whilst the primary driver has been mobiletelephony, the desire to implement mobile data-based services over theseinfrastructures, has led to the rapid development of data-capable bearerservices across such infrastructures. This has opened up the possibilityof many Internet-based services being available to mobile users.

[0143] By way of example, FIG. 2 shows one form of known communicationinfrastructure for mobile users providing both telephony and data-bearerservices. In this example, a mobile entity 20, provided with a radiosubsystem 22 and a phone subsystem 23, communicates with the fixedinfrastructure of GSM PLMN (Public Land Mobile Network) 10 to providebasic voice telephony services. In addition, the mobile entity 20includes a data-handling subsystem 25 inter-working, via data interface24, with the radio subsystem 22 for the transmission and reception ofdata over a data-capable bearer service provided by the PLMN; thedata-capable bearer service enables the mobile entity 20 to communicatewith a service system 40 connected to the public Internet 39. The datahandling subsystem 25 supports an operating environment 26 in whichapplications run, the operating environment including an appropriatecommunications stack.

[0144] More particularly, the fixed infrastructure 10 of the GSM PLMNcomprises one or more Base Station Subsystems (BSS) 11 and a Network andSwitching Subsystem NSS 12. Each BSS 11 comprises a Base StationController (BSC) 14 controlling multiple Base Transceiver Stations (BTS)13 each associated with a respective “cell” of the radio network. Whenactive, the radio subsystem 22 of the mobile entity 20 communicates viaa radio link with the BTS 13 of the cell in which the mobile entity iscurrently located. As regards the NSS 12, this comprises one or moreMobile Switching Centers (MSC) 15 together with other elements such asVisitor Location Registers 32 and Home Location Register 32.

[0145] When the mobile entity 20 is used to make a normal telephonecall, a traffic circuit for carrying digitised voice is set up throughthe relevant BSS 11 to the NSS 12 which is then responsible for routingthe call to the target phone (whether in the same PLMN or in anothernetwork).

[0146] With respect to data transmission to/from the mobile entity 20,in the present example three different data-capable bearer services aredepicted though other possibilities exist. A first data-capable bearerservice is available in the form of a Circuit Switched Data (CSD)service; in this case a full traffic circuit is used for carrying dataand the MSC 32 routes the circuit to an Inter-Working Function IWF 34the precise nature of which depends on what is connected to the otherside of the IWF. Thus, IWF could be configured to provide direct accessto the public Internet 39 (that is, provide functionality similar to anIAP—Internet Access Provider IAP). Alternatively, the IWF could simplybe a modem connecting to a PSTN; in this case, Internet access can beachieved by connection across the PSTN to a standard IAP.

[0147] A second, low bandwidth, data-capable bearer service is availablethrough use of the Short Message Service that passes data carried insignalling channel slots to an SMS unit which can be arranged to provideconnectivity to the public Internet 39.

[0148] A third data-capable bearer service is provided in the form ofGPRS (General Packet Radio Service which enables IP (or X.25) packetdata to be passed from the data handling system of the mobile entity 20,via the data interface 24, radio subsystem 21 and relevant BSS 11, to aGPRS network 17 of the PLMN 10 (and vice versa). The GPRS network 17includes a SGSN (Serving GPRS Support Node) 18 interfacing BSC 14 withthe network 17, and a GGSN (Gateway GPRS Support Node) interfacing thenetwork 17 with an external network (in this example, the publicInternet 39). Full details of GPRS can be found in the ETSI (EuropeanTelecommunications Standards Institute) GSM 03.60 specification. UsingGPRS, the mobile entity 20 can exchange packet data via the BSS 11 andGPRS network 17 with entities connected to the public Internet 39.

[0149] The data connection between the PLMN 10 and the Internet 39 willgenerally be through a firewall 35 with proxy and/or gatewayfunctionality.

[0150] Different data-capable bearer services to those described abovemay be provided, the described services being simply examples of what ispossible.

[0151] In FIG. 2, a service system 40 is shown connected to the Internet40, this service system being accessible to the OS/application 26running in the mobile entity by use of any of the data-capable bearerservices described above. The data-capable bearer services could equallyprovide access to a service system that is within the domain of the PLMNoperator or is connected to another public or private data network.

[0152] With regard to the OS/application software 26 running in the datahandling subsystem 25 of the mobile entity 20, this could, for example,be a WAP application running on top of a WAP stack where “WAP” is theWireless Application Protocol standard. Details of WAP can be found, forexample, in the book “Official Wireless Application Protocol” WirelessApplication Protocol Forum, Ltd published 1999 Wiley ComputerPublishing. Where the OS/application software is WAP compliant, thefirewall will generally also serve as a WAP proxy and gateway. Ofcourse, OS/application 26 can comprise other functionality (for example,an e-mail client) instead of, or additional to, the WAP functionality.

[0153] The mobile entity 20 may take many different forms. For example,it could be two separate units such as a mobile phone (providingelements 22-24) and a mobile PC (data-handling system 25) coupled by anappropriate link (wire-line, infrared or even short range radio systemsuch as Bluetooth). Alternatively, mobile entity 20 could be a singleunit such as a mobile phone with WAP functionality. Of course, if onlydata transmission/reception is required (and not voice), the phonefunctionality 24 can be omitted; an example of this is a PDA withbuilt-in GSM data-capable functionality whilst another example is adigital camera (the data-handling subsystem) also with built-in GSMdata-capable functionality enabling the upload of digital images fromthe camera to a storage server.

[0154] As regards the service provided by the service system 40, thiscan be a location-aware service (also known as a “location-based” or“location-dependent” service), being a service that takes account of thecurrent location of the mobile entity 20. The most basic form of thisservice is the emergency location service whereby a user in trouble canpress a panic button on their mobile phone to send an emergencyrequest-for-assistance message with their location data appended.Another well known location-based service is the provision of trafficand route-guiding information to vehicle drivers based on their currentposition. A further known service is a “yellow pages” service where auser can find out about amenities (shops, restaurants, theatres, etc.)local to their current location.

[0155] Location-aware services all require user location as an inputparameter. A number of methods already exist for determining thelocation of a mobile user as represented by an associated mobileequipment. In addition to location discovery systems based on GPS(Global Positioning System), there exist a number of other systems themost notable of which are those that rely on cellular radioinfrastructures. More particularly, within a PLMN coverage area, it ispossible to get a reasonably accurate fix on the location of a mobileentity by measuring timing and/or directional parameters between themobile entity and multiple BTSs 13, these measurement being done eitherin the network or the mobile entity (see, for example, InternationalApplication WO 99/04582 that describes various techniques for effectinglocation determination in the mobile and WO 99/55114 that describeslocation determination by the mobile network in response to requestsmade by location-aware applications to a mobile locationcenter-server-of the mobile network).

[0156]FIG. 2 depicts the case of location determination being done inthe network, for example, by making Timing Advance measurements forthree BTSs 13 and using these measurements to derive location (thisderivation typically being done in a unit associated with BSC 14). Theresultant location data is passed to a location server 41 from where itcan be made available to authorised services. Thus, when the mobileentity 20 wishes to invoke a location-aware service available on servicesystem 40, it sends a request to service system 40 via a data-capablebearer service of the PLMN 10 and the internet 39; this request includesan authorisation token and the mobile entity ID (possible embedded inthe token). The service system then uses the authorisation token toobtain the current location of the mobile entity 20G from the locationserver 41 (the location server 41 will probably not be holding currentlocation data for the mobile entity 20 and will need to request theappropriate BSC to determine this data before returning it to theservice system 40). The use of an authorisation token is unnecessary ifthe service has been prior authorised to the location service by themobile entity. Of course, as an alternative to having the service obtainlocation data from the location server 41, the mobile entity could haverequested its location from the location server and then included thisinformation in the request to the location-aware service running onservice system 40.

[0157] Whilst the above description has been given with reference to aPLMN based on GSM technology, it will be appreciated that many othercellular radio technologies exist and can typically provide the sametype of functionality as described for the GSM PLMN 10.

1. A digital camera comprising: an image recording subsystem for makingimage recordings; a location-providing subsystem for providing locationdata items representative of a current location of the camera, alocation-stamping subsystem for associating with image recordings takenusing the image recording subsystem, respective location data items,provided by said location-providing subsystem, that each represent thelocation of the camera at the time the corresponding image recording wastaken, and a location recording subsystem for effecting user-triggeredrecording, independently of the recording of an image recording, oflocation data items provided by the location providing subsystem, theselocation data items and those associated with the location-stamped imagerecordings being recorded in a sequential association with each other.2. A camera according to claim 1 , wherein both the location-stampedimage recordings and the location data items recorded independently ofrecording an image recording, are recorded in data structures of thesame form, the data structures being arranged or explicitly linked intoa sequence indicative of the order of recording.
 3. A camera accordingto claim 1 , wherein each location data item recorded independently ofrecording an image recording, is recorded in a log distinct from thelocation-stamped image recordings along with sequence data indicatingits sequence position relative to the sequence of recorded imagerecordings.
 4. A camera according to claim 1 , wherein the imagerecording subsystem and the location recording subsystem haveindependent user-operable controls for triggering recording.
 5. A cameraaccording to claim 1 , wherein the image recording subsystem and thelocation recording subsystem have a common user-operable control fortriggering recording and a common sensor for determining whether thereis an image to be recorded, the image recording subsystem beingoperative to record an image recording when said control is operatedonly if said sensor indicates that an image is present, and the locationrecording subsystem only recording a location data item when saidcontrol is operated, if said sensor indicates that an image is absent.6. A camera according to claim 5 , wherein said sensor is a sensor fordetecting whether a lens shutter of the camera is open or closed.
 7. Acamera according to claim 5 , wherein sensor is an optical sensorlocated in an image plane of the camera.
 8. A camera according to claim7 , wherein said optical sensor is constituted by an electronic imagecapture device of the image recording subsystem.
 9. A camera accordingto claim 1 , wherein the location providing subsystem is a GPSsubsystem.
 10. A camera according to claim 1 , wherein the locationproviding subsystem comprises a cellular mobile radio subsystem forreceiving location-related data from a mobile radio infrastructure.